Skip to main content

Airbyte Connection detailed description

Airbyte Introduction​

Airbyte is the new open-source data integration platform that runs in the safety of your cloud and replicates data from applications, APIs & databases to data warehouses, lakes and other destinations. Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Overview​

This document contains the process to get the data from the various vendors to the AIV application. Let’s understand it with the small diagram: -

Docusaurus Slash Introduction

The Airbyte will collect the Detected datasets and provide curated datasets to our application. For more detail understanding, go to Airbyte documentation: https://docs.airbyte.com/

Airbyte Characteristics​

  • The Airbyte Auto Sync the updates of the data from the vendor as per the selected time for example from once in 5 minutes to every 24 hours.

  • User can sync the data manually every time they update from the vendor.

  • User can add a prefix to streamed sources.

info

  • The Airbyte only takes the data from the Vendor and delivers it to the AIV application.

  • The Airbyte only fetches the data, which is provided by the vendor, we can say that the vendor is responsible for all the data limitations.

  • The Vendor can have data in the format of Excel, Multi-tab Excel, CSV, CSV without headers, Delimited Text, Fixed width Text e.g., 80Bytes, Fixed position text, Log files (grok patterns), EDI, Avro, Parquet, JSON, JSONL, XML, OCR.

FAQ​

  1. What if we connect the source and fetch the data to make datasets in our application then the datasets seem empty? Ans: Yes, it may seem empty, due to not having the data or it may show you the Sample data provided by the vendor. to bypass that situation, try to make an app, project on the vendor’s side.
  2. Why the sync takes much time to get the data? Ans: It depends on the vendor’s logs and its loading time.
  3. What if we have all the properties but the airbyte connection gets failed to connect us with the vendor. Ans: If you have all the properties and the connection gets failed, then you can raise the Question to the Airbyte developers on https://discuss.airbyte.io/new
  4. Can we add a specific time to sync the data on airbyte? Ans: No, There is no option available right now on airbyte, you need to add a specific time duration to sync the data like once a day(Every 24 hours) or every 5 minutes.